Load Balancer health check probes to TCP port 22 causes slow commits
17291
Created On 05/04/21 00:40 AM - Last Modified 02/07/24 04:37 AM
Symptom
In cloud deployments using a Load balancer, to handle health checks, management profile on the DP interface (or loopback) with either SSH or HTTP or HTTPS can be enabled
If the load balancer uses TCP port 22 for health check probes and the VM-Series responds to Health Checks, it may lead to issues like:
1. Higher than expected commit times
2. Longer time for CLI/ GUI access
3. High Management Plane (MP) CPU
Environment
VM-Series Responding to Health Checks
- In these scenarios, VM-Series will respond to the Health-checks
- Management profile feature on VM-Series interfaces can be leveraged for responding to HCs
- Destination IP address on Health Checks dictates where the management profile is added:
- Primary IP address of the interface attached to Load Balancer
- Create a management profile on the interface attached to the Load Balancer
- Forwarding rule IP address
- Create a loopback interface on the interface attached to the Load Balancer and add a management profile on the loopback interface
- Primary IP address of the interface attached to Load Balancer
Cause
It just happens that some load balancers like GCP health checks use multiple probers as mentioned here.
https://cloud.google.com/load-balancing/docs/health-check-concepts#multiple-probers
Maybe with modification of interval and timeout could reduce the number of probes to mgmt plane to be from 5 to 15 connections, but this still takes a toll on mgmt plane resources. SSH launches internal shells for each connection request. HTTP does not have the same overhead for Management Plane as SSH connections do.
Resolution
- Avoid TCP port 22 for health check
- If inevitable use a higher time interval say 30-60s as probe interval
- Use HTTP and direct probes to /unauth/php/health.php (Available in 11.0+,10.2.4+,10.19+)
- In other versions direct probes to /php/login.php
Additional Information
Health Checks Overview